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LINEAR SYSTEMS WITH N ON STATIONARY INPUTS 
By Marvin Shinbrot 

SUMMARY 


By means of examples, a new method is illustrated for optimizing, 
over a finite interval of time, time-varying systems with stationary or 
nonstationary statistical inputs. The method depends on the correlation 
functions being of a certain type, which, fortunately, is the type found 
in a large number of practical problems of importance. 


INTRODUCTION 


Recently, particularly because of their application to missile and 
fire-control systems, considerable attention has been paid to optimiza- 
tion techniques for systems having statistical inputs. Historically, 
such techniques begin with Wiener’s theory (ref. l) of the optimum design 
of constant coefficient linear systems. To apply this theory it is 
required that the statistical properties of both the messages and the 
noise do not vary with time; quantities having this invariance property 
are called "stationary.” This requirement, that messages and noise be 
stationary, is quite restrictive, eliminating from consideration even 
such a simple problem as the optimal determination of the position of a 
gun target, say, when the target is moving with constant speed in a 
straight line and the measurements are corrupted by noise. It seems 
desirable, therefore, to eliminate this hypothesis, if possible. 

In reference 2, Booton attempted to generalize the Wiener theory 
to include nonstationary inputs and time -varying linear operations. 

Booton arrived at an integral equation, the solution of which would be 
the impulse response of the optimum system. However, no method for 
solving the equation was given. 

A new method for solving integral equations such as the one Booton 
derived was recently discovered (unpublished) . This method requires that 
the correlation functions which arise be of a certain form; fortunately, 
this requirement is very frequently satisfied in practice. 

The purpose of this report is the illustration by means of examples 
of the application of this method. Since nonstationary optimization 
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problems may lead to a time-varying linear system as tlie dptimum, the 
report begins with a "brief discussion of the perhaps unfamiliar idea of 
a time-varying transfer function. The report proceeds with an outline 
of a variant of the theory of reference 2, itemized, step-by-step pro- 
cedures for solving the integral equation for the impulse response of 
the optimum system, and some relatively simple examples. Techniques for 
determining the differential equations of the optimum system from the 
impulse response are also shown in the examples. It is hoped that a 
study of these examples will be sufficient introduction to the subject 
to allow the method to" be applied to more difficult practical problems. 


ERKT.TMIMEY DISCUSSION . 
TIME-VARYING TRANSFER FUNCTIONS 


The meaning of the impulse response g(t - t) of a time-invariant 
transfer function is well known. If the transfer function in question 
is of the form 


iisl ■ 

F(p) 

where p denotes the operator d/dt, and f and F are polynomials, 
then g(t - t) is the response of this system at time t to an impulse 
applied when t = r; that is, 

F(p)g(t-r)«f(p) 6 (t-r) 


where 8(t - t) symbolizes the Dirac 8 function (ref. 3). The most 
important property of this inpulse response is that if i(t) is any input 
to the system, then the output in response to i(t) is 



( 1 ) 


For time-varying systems, the idea of an impulse response also 
exists. Consider the transfer function 

f(t,p) 

F(t,p) 

The impulse response g(t,*r) for this system then satisfies the equation 

F(t,p)g(t,T) = f(t,p)8(t - r) 
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It should he noted that for time- varying systems, the Impulse response 
is some function of t and *r, and does not depend on their difference 
alone. The fundamental property (l) carries over to time-varying systems. 
If i(t) is any input to the system, the output is 


/ 


g(t,T.)i(T)dT 


(2a) 


As an example of this, consider the simple system with transfer 
function 


1 

P+t2 

The impulse response satisfies the equation 

~ g(t,T) + t 2 g(t,T) = S(t - t) 
It can he shown that this means 


r 3 -! 3 

g(t,r) = e 3 u(t - r) 

(where u(t) is the unit step function: u(t) =0, t < 0, u(t) = 1, t > 0) 

so that the response to any input i(t) is 

t 3 pt r 3 
e 3 / e 3 i(”r)dT 


Note, incidentally, that 

g(t,*r) = 0 for t < t 

This must always he the case for, if it were not, the system would he 
required to respond to an input before the latter occurred. 


NOTATION AND DEFINITIONS 


Although the ideas discussed herein have application to different 
fields, for ease of expression, the language of communication theorist 
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will "be used throughout - thus, we shall speak of "f: liters” designed to 
derive from an "input" an approximation to a "message" which has "been 
corrupted by "noise." 

Not, there will always be an ensemble of messages it is desired to 
transmit. Denote a typical message by the symbol m(t; a.i, . . ., a^j); the 
parameters a indicate which particular message of the ensemble is being 
considered. Denote the possible noise functions by n(tj 0!, . . ., 0^) . 
The input to the filter we are attempting to design is defined by 


i(t; <x>i, • * •> Qjj* 3 1 > • • •» 3jj) 


— m(tj ai, . . ., a$j) + n(t; 0i, • . Pjj) 


If f(tj ax, . . ., ajj, 0i, . . ., 0]j) is any function of t and 
the parameters a and 0, we shall mean by 


Av 



»i. 


•> “M* 3u 



the average of f with respect to the a*s and 0’s, that is, with respect 
to the ensemble. 

For later use, we now define the following correlation functions: 


“ At ai * * * a i> • • •> 

(~b,T) ~ At ai, • • ., ai, • « • , ctjj, Pi, • • • , 

<Pj i ( t 7 T ) - Av ji(tj °-l, • • • , Pi, • • •> Pjj)t( T J • • •» **11* Plj 
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THE INTEGRAL EQUATION FOR THE OPTIMUM 


In. reference 2, Boot on, using the methods of reference k, derived 
an integral equation for an optimum filter. Although the methods to he 
discussed are applicable to Booton* s equation also, we shall apply it to 
a slightly different one. The reason for this is that Booton considered 
the general case when the inputs to the filter have no beginning in time, 
so that the system has been in operation infinitely long. Of course, in 
all real situations, a starting point exists - a time when the telephone 
is first picked up or the missile is first fired, etc. In view of this, 
the simplifying assumption that the inputs are nonexistent (zero) up to 
a certain time will be made here. By an appropriate choice of the time 
scale, this distinguished instant may be made zero. In this case, the 
response (2) of a system with impulse response g(t,T) to the input 
i(t) reduces to 


/ 

• o 


g(t,T)i(T)dT 


(2b) 


Now, it is desired to filter the inputs to give as good as possible 
a representation of the messages; that is, it is desired to specify an 
impulse response g(t,*r) such that, in accordance with (2b) 



•> Otjj, 




is as close as possible to m(t ; a x , . . . , ajj) . The expressions "as 
good as possible" and "as close as possible" remain to be defined. They 
will be taken to mean that the mean square error is as small as possible - 
that is, g(t ,t) is to be chosen such that 


E 2 (t) a Av 



ai. 


•* 



•) Plj 



(*a> 


is a minimum. It should be noted, incidentally, that the mean square 
error E 2 is a function of t since the average is taken with respect 
to the ensemble with t fixed. 

Now, it is not hard to show that equations (3) and (Ifa) together 


give 
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pt pC pC 

E 2 (t) = cp lDm (t,t) -2 J g(t,r)cp ml (t,T)dT+ J g(t,r) J gCt,<r)q> ±1 (T,<r)aff dr 

o o o 

(5a) 


and, by a method exactly parallel to the one used in references 2 and b, 
it can be shorn that this error is a minimum if and only if g(t,r) 
satisfies the linear integral equation 




g(t,tr)<p 11 (T,tr)dff 


for 0 < t < t 


(6a) 


Inserting equation (6a) into (5a) gives for the minimum mean square 
error: 


ff^ninCt) =qUn(t,t) -J g(t, T )q^ 1 (t,T)dT 


(7a) 


Uot infrequently, it is desired to generalize this technique in the 
sense that it is desired to obtain from the input some quantity dependent 
on the message but not the message itself. Thus, for example, it might 
be desired to predict the message h seconds hence or to integrate or 
differentiate the message. To do this, we minimi ze 


E^t) = Av-j^ |ja(tj • • • t c ^{) " 

/ g(t ’ T) i(T5 ch.> • • •> Qm* Pi/ 





where \i is the quantity we wish to obtain from the input. Thus, if it 
is desired to integrate the message, p(t; oi, . . ., a$,j) is the integral 
of m(t; oi, . . . , ccy); if it is desired to differentiate the message, 
jjl is the time derivative of m, etc. Defining 


v (t,T) = At 



»1, 


., eta. 



< P t ii( t ' T ) = At 





■ } a^iir; ax, . . ., a^. Pi, 
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we find that the generalization of equation (5a) is 



The equation from which the optimum impulse response is to be determined 
is 


n’t 

< P( Jl i( t ' T ) = / gCt^cp^fT^Jda , for 0 < t < t 


(6b) 


and the minimum mean square error is 


r 

=tp [I|1 (t J t) -J gCtj-iOcp^iCtj-iOdT (7b) 


Since equation (6b) is the more general, in the sense that it reduces 
to (6a) when p is set equal to m, all considerations which follow will 
refer to this equation. If the problem is simply one of filtering, it 
will only be necessary to set p = m in all that follows . 


ASSUMPTIONS 


The basic assumptions which will be made in this paper are the 
following: 

(i) The functions cp^Ct,! - ) and cp^ i (t,r) are of the form 1 


r -1 


?li(t;T) - < 


2 j a p ('t)bp(T) , t < t 


^ a p( T )bp(t) , 


t — j 

1 


T > t ) 

<P kli ( t , T ) ■ ^0p(t)bp( T ) , T<t 


(8a) 


1 More generally, cp^ is allowed to involve 5 functions depending 

on t - t (white noise); however, as will be seen, this is a limiting case 
of the form (8) . 
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(ii) If ap(t) and bp(t) have the same meaning as in (i), then the 
quantity 


P 



1 


is a function of t - t alone: 


w = w(t - t) 

The following remarks on these assumptions are pertinent. First, 
in almost all cases of importance, cp^ and m . either will be of the 
form (8a) or may be successfully approximated by functions of this form; 
this can be seen from the definition ( 3 ) of <p^ and as averages of 

products of functions of t and functions of r. Second, note that the 
expression for cp^^t ,t) when t > t is a necessary consequence of the 
assumed form when t < t and is not a separate assumption since, by its 
definition ( 3 ), (p i;L (t,T) = cp^(T,t). Third, under certain circumstances 

where the assumption (ii) does not apply, alternative methods Btill exist. 


EXAMPLES 


Five examples will be given. They ware chosen primarily to illus- 
trate different aspects of the problem of solving the fundamental integral 
equation. In order to do this most successfully, the examples have been 
broken tip into three groups: examples involving white noise, examples 

with other types of noise, and a third class of examples, the necessity 
for which trill appear as we proceed. 


WHITE NOISE 


White noise is said to occur when the autocorrelation function of 
the noise has the form 


tp m (t,T) = N8(t- t) 


where S(t - t) denotes the Dirac 5 function. As will be seen in the 
next section, this kind of noise can be considered as a limiting case of 
"continuous” noise, and so a problem involving white noise can always be 
solved by sett ing up and solv ing another problem involving continuous 
noise and then taking limit s. A simpler procedure, however, would be to 
describe a method for solving problems with white noise directly. 
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In order to do so, some assumptions resembling (i) anfl (ii) above 
must be made. We shall assume the followin g : 

(i*) The functions of cp^(t,T) and cp^(t,*r) have the form 


Q 

^aq(t)bq(T)+ N6(t - t) , 


T < t 


<Pii(t,r) =< 


1 

Q 


aq( r)h q (t ) + N5( t - t) , 

x 

Q 

<P (1 i(t, T ) = ^c q (t)b q (T) , 


>t > 


T < t 


(8b) 


(ii*) If aq(t) and hq(t) have the same mean in g as in (i 1 ), then 
the quantity 


= ^[aq(t)‘bq(r) - aq(>r)bq(t)] 


is a function of t - r alone: 


v = v(t - t) 

With these assumptions, a method for solving the integral equation when 
the noise is white can be outlined. 

STEP 1. Let Bq(s) and V(s) denote the Laplace transforms of 
bq(t) and v(t) , respectively. Then, find the quantities 


n - 1 « 

* N + V(s) 


and let 7 q(t) denote the inverse Laplace transform of Tg(s). 
STEP 2. Compute 

XpqCt) = J ®p( T ) 7q( T )dr 


and solve the equations 
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[1 + Xii('fc)]gx('t) + g 2 (t) + 

l2l(t) 6l(t) + [1 + I22(‘t)]g2('t) + 


. + IlQ(t) gQ(t) = C x (t) 

* + 6q(t) = c 2("fc) 


( 9 ) 


iQi(t) Si(^) + XQa(t) ga('t) + • • • + [! + XQQ('t)]gQ( , t) = CQ(t) 


for the functions gq(t) , q = 1, . . . , Q. 

Finally, let u(t) denote the unit step function: 


ui 


(t) = 


0 , t < 0 

1 , t > 0 


Then, the function 


g(t,r) = u(t-r) \ gg(t) 7 t (T) 


( 10 ) 


satisfies the integral equation for the optimum impulse response. 
We now apply this to two examples. 


Example I 


To begin, consider the following question, equivalent to the one 
already posed in the Introduction. A collection of particles leaves the 
origin at a certain timej thereafter, each particle moves with constant 
(unknown) velocity. Assuming our measurements to he corrupted hy white 
noise, what is the "best way to determine the positions of the particles 
at any succeeding time? 

Calculation of the correlation functions .- Let a denote the speed 

of a typical particle, and let a 2 denote the mean square speed, aver- 
aged over all the particles. Although we do not assume a to he known 
for any particular particle, we do assume the mean square speed a 2 to 
he known. 

We choose the time that the particles leave the origin as zero, so 
that the messages (i.e., the particle positions) are given hy 

m(t;a) = at , t > 0 
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Hence, we can compute 


= Av|(at)(aT)j- 


= a 2 tr 

It is assumed here that the noise is white. It follows that it has zero 
mean. If it is assumed (quite reasonably) that the noise and the message 
are independent, it follows that qp^ = cp ^ = 0, and so 

tp^Ofc,'!-) = Av-|m(t)i(T)j- 

= Av-|m(t)m(T)^j- + Av~j^m(t)n(T)^j- 

“ V(t, T) + <P m (t,T) 

= a 2 tT 


Similarly, 

CP i;L (t,T) =.q> nml (t,T) tep^Ct,^ 

= a a tT + H5(t - t) 

The functions and cp (li (= cp^) clearly have the form (8b) with 

Q = 1. In fact, we may set 

s-i(t) = o^t, b x (t) = t , c x (t) = a^t (3J 


The integral equation and its solution .- To apply the method outlined 
on pages 11 and 12, it is unnecessary actually to write down the integral 
equation (6). However, it is convenient to have this equation, since it 
can be used to check the answer. With the correlation functions given in 
the preceding paragraph, the integral equation becomes 

a 2 tT = a 2 T J «yg(t,(j)dcr + Ng(t,T) , for 0 < r < t (12) 
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The last term In this equation arises from the 6 function occurring in 
the expression for <p^; it is not an integral since the fundamental 
property of the 5 function 



t- cr)dcy = g(t ,t) 


if 0 < t < t 


has teen used. 

We now apply the method outlined earlier to solve equation (12) . 
First, we check to see if assunqption (ii 1 ) is fulfilled. It is, since 

v « aa.(t)b x (T) - a^iOtiCt) 

= a2t * r - a?-r • t 
= 0 


which may he considered as a function of t - r, notahly that function 
which is always zero. 

We now proceed with the solution. 

SEEP 1. Since t x (t) is as given by (ll), we have 


Bi(s) 


_ 1 _ 

s 2 


Also, since 


v(t) = 0, V(s) = 0. Thus, we define 



and inverting 


7iCfc) 


t 

N 


STEP 2. Since in this example, Q = 1, we need only find 
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In(t) = J ax(T) 7 i(T)d 

o 



T 


Hence, equations (9) "become 

(1 + § t s ) gl (t) = 5Jft 

that is, 


gi(t) 




g^t 

1 + (o2/3*0t 3 


Therefore, hy equation ( 10 ), the desired solution of the integral 
equation (12) can he -written: 


g(t,r) 


q g tT 

H + (^/ 3 )t 3 


u(t - t) 


(13) 


It should he noted that g(t,T) vanishes if the impulse to which it 
is the response occurs at the time t = 0 . This is so because we have 
assumed that the initial position of the particles is 'known precisely - 
that is, that at zero time the particles are all at the origin. This is 
one of the many respects in which the present simplified example does 
not describe the true situation for an actual gun platform. 

The result can he checked hy substitution into equation ( 12 ). In 
fact, for 0 < r < t. 
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a a T P crg(t,tr)dtr + Ng(t,r) = a 2 r T 2^5 c^dcr + 

J Q J Q 3ST + (a 2 /3)t 3 W+(a 2 /3)t 3 


. ^ 2 t a, 2 ^; tf + No 2 tr 

N + (a 2 /3)t 3 3 H+ (^2/ 3 ) t 3 

= a 2 tr 


which is as it should he . , 

The error .- The minimum error can he found hy using equation (7a) ; 
in fact. 


E 2 inin(t) 


a 2 1 2 


P a 2 tT 
J Q N +(^/3)t 3 


a 2 tr dT 


= a 2 t 2 


G?ft z t 3 

N + ( a^/3)t 3 3 


N a 2 1 2 
IT + (o^/3)t 3 


Note that the error approaches zero as t —> w. 

The system differential equation .- We shall now derive a differential 
equation which relates the input and output of the optimum system. Thus, 
we seek two functions, f(t,p) and F(t,p), such that 

F(t,p)g(t,T) » f(t,p)8(t - t) (l4) 


where p = d/dfc. These functions are required to he polynomials in p. 
Also, for the impulse response (10) , the order of the differential equa- 
tion is alwayB equal to the number ' of terms in the sum on the right-hand 
side of equation (10) - that is, Q. Thus, we write 
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F(t,p) = pQ + SQ_ 1 (t)p Q " 1 + . ~ + i Q (t) ' (15) 

Since there are no 8 functions in the expression (lO) for g, the order 
of f is at most Q - 1; hence, 

f(t,p) - T)Q_ 1 ( t )P Q ~ 1 + • • • + TJ 0 (t) (16) 

Our problem now is the determination of the functions and 
In example I, Q = 1 , and so 

F(t,p) = p + g Q (t) 
f(t,p) = rj 0 (t) 

Thus, the differential equation (l 4 ) becomes 

g(t,*r) + £ 0 (t)g(t,T) = Tj 0 (t)8(t - t) (17) 

at 

Now, the function £ Q (t) can be found immediately, since S(t - t) = 0 
for t > t. Thus, for t > r we have 


S„(t) « - 


g(t,T) 


Hence, from equation (13) , 


I ( t ) = 


2a g t 3 - 3H 

tCo^t® + 3N) 


and the differential equation ( 17) becomes 


g(t,r) + ? ■ g(t ,t) = ti (t) 5 (t - t) 

St t(a g t s +3N) 


To compute T) Q (t) , substitute g(t,T) as given by equation (13) into (l8) 
This gives 
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5(t - t) = T] (t)b(t- t) 

N + (a 2 /3)t 3 0 

using the fact that (d/dt)u(t - t) = S(t - t) . Now, 


a 2 tr 

H + (^/3)t s 


5(t - t) 


qst 2 

N + (a 2 /3)t 3 


S(t - t) 


since for t f t, hoth sides of this equation are zero, while for t = t, 
they are obviously equal. Hence 


n 0 (t)8(t- t) 


— — s(t - T ) 
N + (a 2 /3)t 3 


that is. 




a 2 1 2 

N + (^/3)t 3 


Therefore, finally, we may say if i(t) is any input to the optimum 
system and x(t) the corresponding output, x(t) and i(t) are related by 
the equation 


(a 2 1 3 + 3N)x + ^ x = 3a 2 t 2 i 

t 

Example XI 


For our second example, we consider an air-to-air missile attack 
situation. Since the purpose of this report is not the development of 
an optimum missile guidance system, the problem will be simplified to the 
point where the features of the method are not obscured by the special 
requirements of the problem. Although the resulting situation is unreal- 
istic, it does still possess some characteristics of interest. 

A two-dimensional situation, wherein the missile and the target move 
in one plane, will be assumed. The missile is fired at a certain time, 
called zero, at which it is assumed the target position is known precisely. 
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A coordinate system can then be fixed, in space in such a way that the 
target displacement initially is zero. It is assumed, that after a cer- 
tain time T the missile is moving so slowly it can no longer do any 
damage; thus, if the target is to take effective evasive action, it must 
do so in the first T seconds after the missile is fired. F inally , it 
is assumed that the evasive maneuver concerned is a step of magnitude 
<xi, the jump occurring at time a 2 . Thus, we are considering the fol- 
lowing ensemble of messages (target displacement) : 


m(t;cca,a 2 ) 


0 , t < ag 

Ot>i , t > <x 2 


Calculation of the correlation functions .- Let us assume a certain 
probability distribution of magnitudes ax is known; thus, we assume 
knowledge of a function f(x) such that the probability that ax lies 
between x and x + dx is f(x)dx. As for the time at which the 

jump occurs, let it be assumed that this quantity has equal likelihood 
^f taking on any value in the interval 0 < a 2 < T. Then, for 0 < t < t, 

<Pnnn(t ,t) = Av |m(tjax,a 2 )m(T;ax,a 2 ) 


+ 00 

f(a-x) 

-00 


Since mC-r;®!,^) = 0 for > t, this simplifies to 

+ 00 t 

=yf f (°l) f a l 2 ^ a 2^ a, 1 

- 00 O 

+ 00 

= jjjJ* f (aiJax^dcta 

—00 

The integral occurring here is just the mean square value of ax, what- 
ever the distribution funct ion may be; consequently, denoting the mean 
square value of a a by ax 2 , we obtain 



T 

/■ 


m(t;ax J a 2 )m(T;ax,a 2 )da 2 dai 
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for r < t 


Now, it is obvious from the definition ( 3 ) that the autocorrelation 
function c p^ is symmetric in t and t: cp^t,!*) = cp^T/fc). Hence, 


< P ram ( t ^ T ) 



for t > t 


Once again, we assume the noise white and independent of message, 
so that 




a-l 
T T 

for 

T < t 

S t 

for 

T > t 


T 


cPiiC-t.T) = cp^Ct, T)+<p nn (t,r) = < 


t + NS(t- t) 
T 


o-i 


t + NS(t- t) 


for t < t 
for t > t 


Comparing this form with equation (8b), we see that assumption (i 1 ) will 
he fulfilled if we choose Q = 1 and 


ai(t) = 


t>i(t) = t 


cx(t) - 


The integral equation and its solution .- Equation (6a) becomes, in 
this case. 


^ T = ^T ~f trg ( t ' ff ) dcr 




+ Wg(t,T) , 


0 < T < t 


Before proceeding to the solution, we check that assumption (ii* ) 
is fulfilled. We have 
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v = a^tJb^T) - ajXrJbjXt) 


.2£ T -*£ t 


a± z 


(t-T) 


Hence it is fulfilled, with v(t) = - t. 
We now apply the method outlined earlier . 


STEP 1. We find first that 


Hence, we define 


Consequently, 


where 


Bi(s) = \ 




fits) = —15= 


Ms* - stf 


(t ) . 

NA 


A 2 
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SEEP 2. We have 


In(t) = / a^T^C^dT 




ai 2 sinh At 
T NX 


cLt 


= cosh At - 1 


Hence, equations (9) become 


gl (t)cosh At = a NX 2 


that is, 

gi(t) = NA 2 seeh At 

Therefore, from equation (10), the optimum Impulse response can 
he written: 

s( t , t) = 

The error .- The mini mum 
(7&) to he 

• ^mint*) - Hjh * 

= NX tanh At 


Au(t - r)sech At sinh At 


mean square error may he found from equation 




ai* 

A sech At sinh At t dT 


The system differential equation . - By exactly the same method as was 
used in example I, the differential equation relating the output of the 
optimum system to the input can he shown to he 


x + (A tanh At)x = (A tanh At)i 
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OTHER TYPES OF NOISE 


In this section, we shall assume the autocorrelation of the inputs 
not to contain 5 functions. With this assumption and assumptions (i) 
and (ii) of pages 7 and 8, the method for solving the integral equation 
can be outlined as follows: 

STEP 1. Let Bp(s) andw(s) denote the Laplace transforms of 
bp(t) and w(t) , respectively, and define 


r p (s) = 


_ Bp(s) 
W(s) 


Let 7 (t) denote the inverse Laplace transform of r p (s). 


STEP 2. Set 


I pg (t) =J a p (r)7q(T)dT 


Let d p (t), p a 1, , , P be any P functions having the form 
cLp('t) = c p (t) - y\ k (t) a p (t) 


i—j 

k 


(19) 


Solve the equations 


[1 + Iu(t) ] gi(t) + Iiz('t) fc^) + « • ♦ + Iip("fc) gp(t) — dx("t) 

I 21 (t) g-i(t) + [1 + Tszi't) ] ga(t) + . . . + Igp(t) gp(t) = flaCt) 

Ipi(t) gi(t) + Ip 2 (t) ga(t) + . . . + [1 + Ipp(t)]gp(t) = dp(t) 


> ( 20 ) 


for the gq(t) as functions of the d p (t) . Then, the function 

P 

g( t , t) =^h k (t)8 ^ k ^(t- t)+ u(t- t) J^g v (t)7 v (T) (21) 

k i 

will satisfy the integral equation (6), whatever the functions h k may 
be . 
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SEEP 3* Choose the functions hjj. in such a way that g(t ,t) 
is as simple as possible, involving the least possible n umb er of 
differentiations . 

It is realized that the description of step 3 is rather vague, but 
examination of the' examples which follow should clarify this . 

We now turn to the examples.' 


Example III 


This third example will be the same as the first, the only difference 
being in the noise. 

The correlation functions .- As in example I, the autocorrelation of 
the messages has the form 


<P nm (t,T) = ® 2tT 


( 22 ) 


Let us assume that the noise is independent of the message and has zero 
mean. Then, as before 


TmiC^T) = cpjnmCt,!-) 


= a 2 tr 


(23) 


Finally, we shall take the autocorrelation of the noise to be 

< Pnn^- ,T ^ = Be”Pl t-T l (2k) 

This expression approximates the autocorrelation of actual radar noise. 
The assumptions male imply that 

<Pj_i(t,r) = o^tT + Be“Pf t-T ^ 

We mention here that the noise described by equation (2k) approaches 
white noise as B and p approach infinity; this will be shown belc5w. 
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The integral equation and its solution .- Once again, it is not neces- 
sary to write down the integral equation, but we shall do so since it can 
be used to afford a useful check on the results. With the correlation 
functions of the preceding section, the integral equation (6a) becomes 


a 2 tT = a 2 T 






+ 



0 < T < t 


(25) 


We now solve this equation by the method out lin ed - Comparison of 
the expressions (22), (23), and (2^) for the correlation functions with 
the expressions (8a) gives the results P = 2, and 


a i(t) = a 2 t b x (t) = t c x (t) = a 2 t 

a 2 (t) = Be-P* b 2 (t) = eP* c 2 (t) » 0 

Hence, 

w = o2t • t - aFr * t + Be"P t • eP T - Be”P T • eP* 

= -2B sinh p(t - t) 

= w(t - t) 

Thus, assumption (ii) is satisfied with 
, w(t) = -2B sinh pt 

STEP l. Taking Laplace transforms, we have 


Bl(s) = i , =,(■) = 





Therefore, we define 


IiCs) = - 


s 2 - 3 2 1 _ 
2Bp s 2 



r ( B ) = 82 - P 2 1 . 

^ ' 2B3 s - p 

s + 3 
2B3 

Inverting the Laplace transform gives 


7l(t) « 35? te 2 * - s(t)l 


7a(t) - [8(t) - ps(t)] 


STEP 2. We have 

Iix(-t) = j|g£ J t[3 2 t - S(T)]dT 
o 

- P 02 +3 

" 6b 

Iin(t) = ” J t[6(t) - 35(T)]dT 

o 

S 2 

" 2B3 



(r)dr = (-l) n f (n) (0) 


since, in general. 
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l2i(t) = ^J' e“ pT [p 2 T - 5 (t)]cLt 
o 

= ^{ [1 " e " Pt(l + Pt)] ‘ X } 


= _ e -pt 1 + Pt 


^(■t) e“ PT [5(T) - P5 (t)]cLt 


= ^(P-P) 


Consequently, equations (20) for g x and g 2 become 


( x + t3 ) gi(t) ■ iff* " di(t) 


_®_pt L±J£ gl (t) + g2 (t) = a 2 (t) 


These equations can be solved to give 


gx(t) = 6p 


g 2 (t) = 2p 


2Bpd 1 (t) + q2 d 2 (t) 

2p 2 (6B + po^t 3 ) - 30^(1 + pt)e“P^ 

3B(l + pt)e"P t d 1 (t) + p(6B + pqgj^)d 2 (t) 
2p 2 (6B + p a 2 1 3 ) - 352(1 + pt)e -pt 


(27) 



26 


MCA TN 3791 


STEP 3* We have now arrived at the task of solidifying the earlier 
vague description of this step. The idea behind step 3 is very simple 
and was conceived in order to reduce the number of differentiations of 
noisy inputs. 

Note first that if all functions h k were taken to be zero, 
the solution (21) would contain only the second sum in that expression 
and so would resemble very closely our earlier solution (10) for the 
case of white noise. However, although the solution (10) of any prob- 
lem involving white noise will never contain a differentiator, 2 this 
is not the case for other types of noise. In the present example III, 
the existence of a differentiator manifests itself through the occurrence 
of a term 6(t) in the expression (26) for 7 2 (t) . 

Now, the function 7 2 (t) is multiplied in equation (21) by g 2 (t) . 
Consequently, if g 2 could be made to be zero, the differentiator in 
7 2 would be eliminated. From equation (27), we see that setting g 2 

equal to zero gives an equation in the d’ s . According to equation (19), 
however, the d* s are functions of the h’ s. Therefore, if the func- 
tions hfc are free and may be assigned at will, there is the possibility 
that g 2 (t) may be made to vanish. In this example. Only one function 
(g 2 ) must be eliminated; consequently, there appears to be need for only 
one function h, and so we set 

h k (t) = 0 , k > 1 

leaving only ho(t) . J . 

Now, setting g^t) equal to zero gives 

3B(l + pt)e _ P t d 1 (t) + p(6B + po2t 3 )d 2 (t) * 0 (28) 


By virtue of equations (19) and the given values of the functions 
a n (t) , we have 


di(t) = ^t[l - ho(t)] 

d 2 (t) a -Be~P t h 0 (t) 

Therefore, equation (28) iB equivalent to 

30^(1 + pt)[l - ho(t)] - p(6B + p^t 3 )ho(t) = 0 


2 This follows from the fact that when the noise is white, the 

integral equation is of the second kind (cf. ( 12 )). 
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that is, 


ko(t) 


3a 2 t(l + pt) 
6Bp + 3a2t +3^ipt 2 + 


We know now that with this choice of ho and with all other hjj. 
taken as zero, g 2 will he zero and the filter will contain no 
differentiators . 

The calculated value of ho when inserted into the expressions 
for d x and d 2 gives ' 


di(t) = 


Pa 2 t(6B + gqSt 3 ) 

6Bp + 3a- 2 t +3a, 2 pt 2 + a 2 p 2 t 3 


d 2 (t) » 


3Ba 2 te~P t (l + pt) 

6Bp + 3<x2t +3a 2 pt 2 + a^p2t 3 


Therefore, 


gx(t) = 


6Bpa 2 t 


6Bp + 3a 2 t + 3a 2 pt 2 + a. 2 ^ 3 
g 2 (t) = 0 

By equation (2l) , then, the optimum impulse response will he 


(29) 


g(t,T) 


gl(t) 

2Bp 


[ p 2 *ru(t - t)+(1 + pt)6(t - t) - 8(t)] (30) 


where gx(t) is given hy equation (29) . 

The error . The minimum error can he found -using equation (7a) ; in 


fact , 
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E g Tn^(t) = a 2 t 2 - — 3 r T[l+pt)8(t-T)+p 2 T-5(r)]a.T 

6Bp43a 2 t+3a 2 pt 2 +a 2 p 2 t 3 J 


= a 2 t 2 -fl = — 2g£ Pfc(l+pt) 

l 6 BB-t3ct 2 t-t5a 2 Bt 2 +a? fi 2 t 3 L 3 JJ 


6 B 3 a 2 1 £ 


6Bp+3a?t43pa 2 t 2 +a^B 2 t £ 


— as t -» on 
pt 


The system equations .- Let us partition the impulse response (30) 
into two parts; we do this because the term 5(t) which occurs is essen- 
tially different from the others. Thus, we write 

g(t,r) = k(t ,t) —1 (t ,t) 


where 


k(t,*r) * gi(t)[p 2 Tii(t-r)+(l+pt)6(t-T)] 


l(t,T) =gi(t).8(r) (31) 


Each of these terms is an impulse response in its own right. We begin 
this section by finding the differential equation satisfied by the 
response k. Thus, we once again seek two functions F(t,p) and f(t,p), 
polynomials in p = d/dt, such that 


F(t,p)k(t,r) = f (t,p)8(t-r) 


The order of F(t,p) is always equal to the number of terms in the second 
sum of equation ( 21 ); thus, in general, this order would be P. However, 
for this example, g^t) =0, and so there is only one term left in this 
sum. Therefore, we set 


F(t,p) = p+£ 0 (t) 
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To find T| 0 and t^, substitute equation (31) into (32) with g 
given by (33) . This gives 

gi(t)[p(l+0T)5(t - r)+(l + pt)6(t - t)] =Ti Q 5(t- t) +Ti 1 (t)&(t- t) 
Hence, just as in example I, 

iloC*)- 3(1+ P'fc)gx('fc) 

Tl 1 (t) = (l + pt)gi(t) 

Thus, the input -output relationship for that part of the optimum system 
described by the impulse response k(t,T) can be written 



30 


NACA T» 3791 


X-li^X 

gi(t) 


(l + 3t)gi(t) 



( 3 ^) 


where gi(t) is given "by equation (29) . 

It remains to discuss the Impulse response 2 (t,*r), This response 
corresponds to a '‘memory" element, the output of which is put through a 
time-variable "gain." This is clear, for according to equation ( 2 ), the 
response of the system with impulse response l to any input i is 



= gi(t)i(0) 


Thus, we find the optimum system to be that system whose response 
is the difference of the responses of the system described by equation (33) 
and ttye "memory- gain" system having the impulse response l . 

The limiting case .- It was mentioned earlier that the noise described 
by the autocorrelation ( 24 ) is approximated by -white noise when B and p 
are large. To see this, consider the integral equation (25). It is not 
hard to show that as 0 00, 

*b V 

J e ’P( T " (J )g( t ,ff)d(j=| g(t, T )+ 0 (^) 

o 

Hence, if B = Nb/ 2 , the last two terms of the integral equation (25) 
approach Ng(t ,t) as p — 5* 00, so that the integral equation itself resem- 
bles equation ( 12 ) more and more closely. This fact can be used as an 
additional check on our results, for the impulse response (30) should 
approach (13) as p — $>• ». This is the case, for with B = Hp/ 2 , then as 
P — > 00, 


gi(t) 3 cP-t JL_ 

2B0 3S + &1? P 2 

Hence, all the terms in equation (30) except the first approach zero, so 
that 


g(t,r) - 3 v 

which agrees with equation (13) . 


3a 2 tT 
3N + a^t 3 


u(t - t) 
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Example XV 


For our fourth example, we consider a problem in approximate differ- 
entiation. This will serve to illustrate the solution when the more 
general formulation leading to equation (6b) is used. The problem, will ' 
be to find the slope of a measured line passing through the origin. 

Calculation of the correlation functions .- Let the equation of the 
line be 


m = at 


where it is assumed that the slopes a have a certain probab ilit y dis- 
tribution. We shall write a 2 for the mean square value of a. Since 
it is desired to find the slope of the lin e, the output of the filter 
should as closely as possible approximate the quantity 


dm. 


About the noise, it will be assumed it is independent of the message 
and described by the autocorrelation function (2^) . Then 


c P^( t ' T ) = Av j® * “j- 


= a 2 


< P | 1 i( t > T ) = Av ja, • o,tJ- 


= 0.2 


a^T 


cpii(t, T ) =Av -jot ■ aTj-+Be"P( t-T ) 

= a?tT + Be“Pl t ~ T l 


for t,T > 0. 

The integral equation and its solution .- Substituting the above cor- 
relation functions into equation (6b), we see that the optimum impulse 
response must- satisfy 
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+ Be“P T 



,a)djo + 



0 < t < t 


(35) 


This equation is very s imilar to the earlier equation (25) and, in 
fact, it can he solved Immediately hy using the solution- (30) of (25) . 
Multiply equation (35) through hy tj the resulting equation can then he 
■written 



which is exactly the same as equation (25) with g(t,cr) replaced hy 
tg(t,a) throughout. Consequently, from equation (30) , the impulse 
response satisfying (35) is 

g(t,-r) " CP S ' r ^C'fc - t)+( 1 + pt) 5 (t - t) - 5 (t)] (36) 


where gi(t) is given hy equation ( 29 ) . 

The error .- According to equation (7b), the mean square error of the 
optimum system will he 


% 2 nHn (' fc ) “ 



6BPO, 2 

6Bp + 3a? t + 3a?pt 2 + a?p 2 t 3 

Note that this implies that the error can he made as s mall as desired hy 
measuring the line for large enough t. 
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The system equations .- As in example III, the optimum response is 
now divided in two parts: 

g(t,*r) = k(t,-r) - Z (t ,t) 


where 


k(t,r) = C P 2 TjjL(t - T)+(l + pt)8(t- t)] 

l(t ' T) “ Issr 6(t) 


The input-output relation corresponding to the impulse response k 
can be found by using the techniques previously described or by using 
equation (3^) . There results 



(l+pt)gi(t) 



As before, the impulse response l corresponds to a memory-gain 
element . 

The limiting case .- The above solution (36) simplifies if B and p 
are large. In this case, set B = Wp/2 and let p — >• 00. Then, 


g(t,T) 


. 3 gl- u(t-x) 
3N +a 2 t 3 


THE CASE w = 0 


It is clear that one special case exists for which the method of the 
last section fails. This is the case when the function w of assump- 
tion (ii) is identically zero. The failure arises in the very first step, 
since the functions Tp(s) cannot even be defined. A technique applicable 
when w = 0 will now be described. 

In this case, unfortunately, there is no method which can be followed 
mechanically in all cases. The reason for this is that if w = 0, there 
may not be any. solution. In fact, it. can be shown that w s 0 implies 
that there is either no solution at all or else there are infinitely many. 

If no solution exists, it means, of course, that the original ques- 
tion was improperly posed and it was not an optimization problem at all. 
Hence, we shall assume that solutions exist. 
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To show how to find a solution, let 


r 

tPi ± (t ^ t) =^ap(t)hp(r) , 
1 
P 

<P|ji = ^ c p(^)^ l p( T ) > 


T < t 


T < t 


as before. We now assume that P functions 7p(t) can he found such 
that the deter minant which has 

Ipq(t) = J ap(h)7q(tT)dcy 
o 

in its pth row and qth column is different from zero. Unfortunately, 
there are no set rules for determining these functions 7 n - If these 
are known, however, the optimum can he determined as follows. 

Solve the following equations for the functions g^(t) : 

P 

^TlpqCiOgqOt) = oLp(t) , p = 1, . . ., P (38) 


Then, the function 

P 

g(t,r) = Vkk(t)6^(t- r)+ n(t - t) ^gp(t)7 p (T) (21) 

P=i 

satisfies the integral equation provided the functions dp(t) are given 
by the equation 

3p(t) = Cp('t) ”^tk('t) a p(^) > p = l, ...,P (19) 

k 


Example V 

To illustrate the methpd just described, consider the somewhat arti- 
ficial problem of measuring the lengths of a collection of rods. Suppose 
it is definitely known that these lengths are not less than A x nor more 
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than A 2 inches, and that our measuring device is a ruler which can he 
read with an accuracy of ±B inches; in this case, the messages, that is, 
the lengths of the rods, can he .interpreted as steps applied at time zero, 
of magnitude a, where A x < a < A 2 . Similarly, the noise, that is, the 
least reading of the ruler, can he interpreted as a step of magnitude 3 
where -B < 3 < B. When viewed in this light, the problem is amenable to 
the techniques of this report. 

The correlation functions .- Assume that all values of a and 3 in 
their respective ranges are equally likely. Then, for t,T >0, 


9mm(t, T ) = Av jm(t;a)m(T;o,)j- 
= Av-|a • aj- 




£2 


a 2 da 


= A x 2 + AjAg + A 2 2 

3 


Setting A 2 = A x 2 + A^Ag + Ag*, we have 


<Pmm('fc>T) — 


A 2 


Assuming a and 3 independent, we see that 
cp m 1 ( t ' T ) »Av-^a(a + 3)j- 


= Av 

3 


w 


«p ±i (t ,t) ® Av -j(a + 3)(a + p)j- 


A 2 Bf 

3 3 


(39) 


(IK)) 
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The integral equation and Its solution .- Substituting the functions 
(Pj^ and cp ii into equation (6a) gives for the integral equation for the 
optimum 

— = ~ Z+ y Z J g(t,a)dcr (4l) 

o 

which clearly is satisfied hy 

g(t,*r) = g 5(t - t) (42) 

Ar +3* 

Thus, a solution of equation (4-1) can he found hy inspection; we 
now attempt to find a solution hy the method described on page 34. In 
order to illustrate a difficulty which sometimes arises when applying 
any of the methods which have been described, we shall begin with a 
wrong procedure. 

Comparing equations ( 37 ) with equations ( 39 ) and (40), we conclude 
that we may write P = 2, and 

*i(t) = Y , bi(t) = 1 , Cx(t) = ^ 

ae(t) = ~ , b 2 (t) * 1 , c 2 (t) = 0 

However, with this choice of these functions, no solution appears to 
exist, for no matter what y x and y s may be 


Ill 1 12 


\ f 7i(T)dT ~ r 7 2 (T)dr 

^ U ^ J 



O O 

121 122 


\ f 7i(T)dT r 



0 0 


Thus, we might conclude from what has gone before that no solution 
exists; however, thiB apparent nonexistence of a solution does not with- 
stand close analysis, since we know that there is at least one solution - 
given by equation (42) . The difficulty encountered is due to the fact 
that the functions a x (t) and a 2 (t) as chosen above are linearly depend- 
ent. In order to arrive at a correct solution, it is always essential 
that the functions a x (t) and a^t) as well as the functions b x (t) 
and b 2 (t) be chosen so that they are independent. 
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Perhaps the simplest correct solution of the problem results by- 
taking P = 1 and setting 

a i(t) = , b x (t) = 1 , Cl (t) = ^ 

Then, there is only one function Ipq, notably 

In(t) = ^ -- -- J 7i(ff)dcr 
o 

From equation (38) , we then conclude that whatever y x may be, 

gl(t ) Mil 

r\ lMaa 

3 J o 

= A a - (A 2 +B 2 )ho(t) 

(A 2 + B 2 ) J 7i(o)dff 
o 

by equation (20). For the usual reasons, we now set hj^t) = 0, k > 1, 
to obtain from equation (2l) 

g(t,T) = ho(t)6(t- t) +u(t- t) — — 7i( T ) ^3) 

(A 2 + B 2 ) [ 7i((r)dn 
o 

This function optimizes the system whatever ho(t) may be; clearly the 
choice of ho(t) which most simplifies g(t,T) is 


to(t) = 


A 2 + B‘= 


which results in 


s(tA) 


in accordance with equation (42). 
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The error .- According to equation (7a), the mean square error of the 
optimum system is 


EW*) - f '/ »(* - ->]f 

O 

= B 2 1 

3 1+ (b/A) 2 

~ if J«i 

3 * A 

It should "be noted, incidentally, that it can he shown that all 
systems described by equation (43), no matter what ho(t) may be, have 
the same error; of course ^ this was implied previously by calling all 
functions (1*3) "optima." To see this, consider the error corresponding 
to the impulse response (4-3). We have 


E2(t) 


= Af Af r 1 
3 " 3 J 


ho(t)a(t- r) +u(t- t) Aa -(A 2 + B^)ho( t ) _ 7i(t) 

(A 2 + B 2 ) J 7i(d)dtr 


dr 


A 2 A 2 
3 3 


(A a +B 2 ) J' 7x(d)dc r ° 


-ffh (t) 


A 2 


A 2 + B 2 



B 2 1 
3 1+(B/A) 2 


~ E 2 min(^) 


The system equation .- It is obvious that the system with impulse 
response (42) is a simple gain. Thus, the output is obtained from the 
input by multiplication of the latter by the gain A 2 / (A 2 + B 2 ) . To find 
the best estimate (according to the present criterion) of the lengths of 
the rods, the measured lengths should be m u l t iplied by A 2 /(A 2 +B 2 ). 
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This result, that the optimum system is simply represented by a gain, 
is quite general, being true whenever the ratio cp^/cp^^ is a constant. 
In such a case, the optimum system is a gain adjusted so that the mean 
square of the output is the same as that of the desired quantity jx. 
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